Papers by Jung Jun Lee
Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | A sparse Mixture-of-Experts architecture has emerged as a highly scalable solution for instruction tuning. |
| Approach: | They propose a mixture-of-Clustered-Experts (MoCE) architecture that allows expert specialization . they evaluate the mechanism on a set of benchmarks and show its superiority . |
| Outcome: | The proposed approach outperforms existing models and benchmarks on instruction tuning scenarios with significant input heterogeneity. |